Chore: Bake DOT into an Airflow worker image - #121

Description

This PR addresses the Airflow worker integration issues described in the technical documentation

Manual Airflow Worker Setup

Previously, the DOT Dockerfile was a standalone Python image, and the Airflow compose stack utilized a discrete, standard Airflow image. To run DOT within Airflow, operators had to manually execute cd /app/dot && ./install_dot.sh inside the running airflow-worker container. Every time the container was downed or recreated, all DOT requirements had to be manually reinstalled from scratch.

Resolution

• Base Image Update: Refactored docker/dot/Dockerfile to strictly derive from apache/airflow:2.2.4-python3.8, maintaining parity with the other Airflow components.
• Pre-baked Dependencies: The Dockerfile now explicitly jumps to USER root to install OS-level postgresql-client dependencies, then reverts to USER airflow to execute install_dot.sh. The DOT Python packages are now installed during the image build process rather than at runtime.
• Docker Compose Enhancements: Updated the airflow-worker service in docker-compose-with-airflow.yml to directly build and use this new dot-airflow-worker:latest image.
• Dependency Pinning: Hardened install_dot.sh and requirements_dot.txt to pin compatible versions of pip, setuptools, and wheel to prevent build failures in the Python 3.8 environment.
  This streamlines operational deployments. The environment is now immediately ready to execute DOT natively via a BashOperator without repetitive, manual setup steps.

In the Technical Documentation, this is addressed in Section 12 .

Asana Task

Deployment Readiness*

Testing

Describe or check:

• Created or updated unit, feature, and/or integration tests
• Typical manual testing in the local env browser, dev pipeline, etc.

Deployment Notes

Describe or check:

• No special deployment steps required

Rollback Plan

Describe or check:

• Standard revert is sufficient (git revert)

Reviewer Guidance / Questions*

Screenshots / Testing Evidence*

SOC 2 Change Management Checklist

• None of the below are true in this code
• New roles/permissions are introduced without review and approval by the product manager
• Hardcoded credentials, secrets, or API keys are present in this code
• Secrets are being managed outside of the approved secrets management process (e.g., GitHub Secrets, environment variables)
• PII or sensitive data handling is introduced or changed without being reviewed against our data classification policy
• Sensitive data is written to logs
• Input validation and sanitization is missing
• An unnecessary attack surface has been introduced (e.g., unused endpoints, open ports, debug modes left enabled)
• Common vulnerabilities have been introduced in the code (inc. any dependencies added or updated)
• No review for common vulnerabilities has been conducted
• Not tested in a non-production environment
• Breaking changes to existing APIs or integrations with downstream consumers being notified
• Performance impact has not been considered or acceptable
• Appropriate audit logging is missing for any security-relevant actions introduced by this change
• Log entries contain sensitive or PII data
• All existing tests do not pass locally (./vendor/bin/pest)

Provide justification if you are submitting a PR with any boxes checked other than the first.

Reminder for Reviewers: By approving this PR you are confirming that you have reviewed the code for correctness, security, and compliance with our engineering and SOC 2 standards. Do not approve PRs where SOC 2 checklist items are checked without documented justification.

*Optional

• To see the specific tasks where the Asana app for GitHub is being used, see below:
  • https://app.asana.com/0/0/1217243670529352